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[0001] TITLE 

[0002] IMAGE PROCESSING DEVICE AND IMAGE PROCESSING PROGRAM 

[0003] CROSS REFERENCE TO RELATED APPLICATION 

[0004] This application claims benefit of Japanese Application No. 2002-204364 

filed in Japan on July 12, 2002, the contents of which are incorporated by this reference 
as if fully set forth. 

[0005] FIELD OF INVENTION 

[0006] The present invention relates to an image processing device and an image 
processing program for generating color digital images. 

[0007] BACKGROUND 

[0008] With the single-sensor image-pickup system employed in digital cameras 

or the like, a single-sensor image-pickup device wherein a different color filter is 
mounted on each pixel is employed, so the output image from the image-pickup device 
has only one color component for each pixel. Accordingly, a color digital image having 
tri-color components for each pixel is generated by performing color processing for 
generating the color information by estimating non-existent color component for each 
pixel. In the same way, with the double-sensor image-pickup system, or triple-sensor 
pixel spatial offset image-pickup system, there is also the need to perform color 
processing for estimation of non-existent color components for each pixel. 
[0009] With this color processing, deterioration such as blurring or false colors, or 
the like, could be caused in a color image finally obtained, unless a suitable method is 
used. Accordingly, conventionally, various color processing methods have been 
proposed. The color processing can be roughly classified into two types; processing 
based upon edge detection, and processing based upon color correlation. 
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[0010] SUMMARY 

[00 1 1] An image processing device according to the present invention comprises: 

a combination average calculation unit for making a combination of two or more pixels 
from multiple pixels having the same color component near a pixel of interest and 
calculating the average for the combination of the color components of two or more 
pixels for multiple kinds of combinations of pixels in the region near the pixel of 
interest; a color correlation estimation unit for estimating color correlation which is a 
correlation between different color components within the region near the pixel of 
interest; and a combination selection unit for selecting one of the multiple combination 
averages calculated by the combination average calculation unit, as the non-existent 
color component for the pixel of interest, based upon the color correlation estimated by 
the color correlation estimation unit. 

[00 1 2] This feature and advantages of the present invention will become further 

apparent from the following detailed explanation. 

[0013] BRIEF DESCRIPTION OF THE DRAWING(S) 

[0014] Fig. 1 is a block diagram which illustrates a configuration of a digital 

camera according to a first embodiment of the present invention; 

[0015] Fig. 2 A is a diagram which illustrates a region near the pixel of interest 

used for the combination average generating circuit according to the aforementioned 

first embodiment; 

[0016] Fig. 2B is a diagram which illustrates the region near the pixel of interest 

used for the color correlation calculation circuit according to the aforementioned first 
embodiment; 

[0017] Fig. 3 is a flowchart which indicates the color correlation estimation 

processing performed by the color correlation calculation circuit according to the 
aforementioned first embodiment; 
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[0018] Fig. 4 is a flowchart which indicates the combination selection processing 

performed by the combination selection circuit according to the aforementioned first 
embodiment; 

[0019] Fig. 5 A is a diagram for describing an example of an edge and generated 

pixel value in the processing performed by the combination selection circuit according 
to the aforementioned first embodiment; 

[0020] Fig. 5B is a diagram for describing an example of an edge and generated 

pixel value in the processing performed by the combination selection circuit according 
to the aforementioned first embodiment; 

[0021] Fig. 6 is a flowchart which indicates the R/B generating processing 

performed by the R/B generating circuit according to the aforementioned first 
embodiment; 

[0022] Fig. 7A is a diagram which illustrates an example of the region used for 

the processing performed by the R/B generating circuit according to the aforementioned 
first embodiment; 

[0023] Fig. 7B is a diagram which illustrates another example of the region used 

for the processing performed by the R/B generating circuit according to the 
aforementioned first embodiment; 

[0024] Fig. 8 is a flowchart which indicates the software processing performed by 

the computer according to the aforementioned first embodiment; 

[0025] Fig. 9 is a block diagram which illustrates a configuration of a digital 

camera according to a second embodiment of the present invention; 

[0026] Fig. 10 is a diagram for describing the processing performed by the 

combination average generating circuit according to the aforementioned second 

embodiment; 

[0027] Fig. 11A is a diagram which illustrates the region used for the region 

judgment circuit according to the aforementioned second embodiment; 

[0028] Fig. 1 IB is a diagram which illustrates the region used for the region 
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judgment circuit according to the aforementioned second embodiment; 

[0029] Fig. 11C is a diagram which illustrates the region used for the region 

judgment circuit according to the aforementioned second embodiment; 

[0030] Fig. 12 is a flowchart which indicates the region judgment processing 

performed by the region judgment circuit according to the aforementioned second 

embodiment; 

[0031] Fig. 13 is a flowchart which indicates the color correlation estimation 

processing performed by the color correlation calculation circuit according to the 
aforementioned second embodiment; 

[0032] Fig. 14 is a flowchart which indicates the combination selection processing 

performed by the combination selection circuit according to the aforementioned second 
embodiment; 

[0033] Fig. 15A is a flowchart which indicates the software processing performed 

by the computer according to the aforementioned second embodiment; 

[0034] Fig. 15B is a flowchart which indicates the software processing performed 

by the computer according to the aforementioned second embodiment; 

[0035] Fig. 16A is a diagram for describing color processing based upon 

conventional edge detection; 

[0036] Fig. 16B is a diagram for describing color processing based upon 

conventional edge detection; 

[0037] Fig. 16C is a diagram for describing color processing based upon 

conventional edge detection; 

[0038] Fig. 17 is a diagram for describing color processing based upon 

conventional color correlation; 

[0039] Fig. 18 is a diagram for describing color processing based upon 

conventional color correlation; and 

[0040] Fig. 19 is a diagram for describing color processing based upon 

conventional color correlation. 
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[0041] The accompanying drawings, which are incorporated in and constitute a 

part of the specification, illustrate presently preferred embodiments of the invention, 
and together with the general description given above and the detailed description of 
the preferred embodiments given below, serve to explain the principles of the invention. 

[0042] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S) 

[0043] Description will be made regarding the embodiments of the present 

invention with reference to the drawings. 

[0044] In prior to the description of the invention, prerequisite technology for 

the invention will be described. 

[0045] With the single-sensor image-pickup system employed in digital 

cameras or the like, a single-sensor image-pickup device wherein a different color 
filter mounted on each pixel is employed, so the output image from the image- 
pickup device has only one color component for each pixel. Accordingly, a color 
digital image is generated by performing color processing for generating the color 
information by estimating non-existent color component for each pixel. In the same 
way, with the double-sensor image-pickup system, or triple-sensor pixel spatial 
offset image-pickup system, there is the need to perform color processing for 
compensation due to the lack of color components for each pixel. 
[0046] With the color processing, deterioration such as blurring or false colors, 

or the like, could be caused, unless a suitable method is used. Accordingly, 
conventionally, various color processing methods have been proposed. The color 
processing can be roughly classified into two types: processing based upon edge 
detection, and processing based upon color correlation. 
[0047] An example of the technique based upon edge detection is an 

arrangement disclosed in Japanese Unexamined Patent Application Publication No. 
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8-298669. Description will be made regarding the technique described in this 
Publication with reference to Figs. 16A, 16B, and 16C. Figs. 16A, 16B, and 16C are 
diagrams for describing color processing based upon conventional edge detection. 
[0048] In a case that the single-sensor image-pickup device has primary-color 

Bayer-array color filters as shown in Fig. 16A, for example, with the pixel of interest 
X having the R component or B component (which is denoted by R0 in the example 
shown in the drawing), the difference between the G components of the left and 
right pixels thereof dH = |G2 - G3| and the difference between the G components of 
the upper and lower pixels thereof dV = |Gl - G4| are calculated, and the average of 
G components, aH = (G2 + G3)/2 or aV = (Gl + G4)/2, is calculated in the direction 
wherein the difference is smaller than the other, thereby estimating it as the G 
component non-existent in the pixel of interest X from the average, as described in 
Fig. 16B. 

[0049] As described above, with the conventional technique based upon edge 

detection, the spatial correlation between pixels is obtained for multiple pixels 
having the same color component near the pixel of interest, and selects the 
combination of the pixels near the pixel of interest with high spatial correlation for 
the same color, thereby generating the same color component non-existent in the 
pixel of interest. 

[0050] On the other hand, an example of the technique based upon color 

correlation is an arrangement disclosed in Japanese Unexamined Patent 
Application Publication No. 11-215512. Description will be made regarding the 
technique described in this Publication with reference to Figs. 17 through 19. Figs. 
17 through 19 are diagrams for describing color processing based upon the 
conventional color correlation. 
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[0051] As shown in Fig. 17, first of all, let us say that a linear correlation 
holds between the color component values Vp and Vq of two given color components 
p and q formed by the color filters on the image-pickup device (e.g., in a case of the 
primary-color filters, the color components are r, g, and b) around the pixel of 
interest, as represented by Expression 1 using coefficients otqp and Pqp. 
[0052] Vq = aqpx Vp + Pqp ...(1) 

[0053] Subsequently, in the region U set around the pixel of interest X, the 

pixels within the region U are classified into three subsets Ur, Ug, and Ub, based 
upon the obtained color components r, g, and b, and the aforementioned coefficients 
ocqp and pqp are estimated based upon the average Ac (c indicates r, g, or b), and 
the standard deviation Sc (c indicates r, g, or b) for the pixel values in each subsets, 
as shown in Expression (2). 

[0054] ocqp = Sq/Sp, pqp = Aq - aqp x Ap ...(2) 

[0055] Expression (2) is obtained based upon an assumption that the ratio of 

the variation of the color component q (standard deviation Sq) as to the variation of 
the color component p (standard deviation Sp) is the gradient aqp for the linear 
correlation shown in the aforementioned Expression 1, and furthermore, the 
straight line with the gradient aqp passes through the point plotted with the 
average Ap for the color component p and the average Aq for the color component q. 
[0056] In this case, as shown in Fig. 18, the estimation values of the 

coefficients differs depending upon the position of the region U around the pixel of 
interest, and particularly, in the event that the region U contains color edges, 
estimation precision is greatly reduced. 

[0057] Accordingly, multiple kinds of regions Uk are set around the pixel of 

interest, and the estimation values for each region Uk are weighted with the 
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reliability estimated from the maximal value of the standard deviation in the Uk, or 
the like, so as to obtain the final estimation values aqp and pqp for color correlation 
with regard to the pixel of interest X, thereby obtaining the value Xm of the color 
component m, which is non-existent in the pixel of interest, from the value Xe of the 
color component e obtained in the pixel of interest X, based upon the estimated aqp 
and Pqp using the above-described Expression 1, as shown in the following 
Expression 3. 

[0058] Xm = ocme • Xe + pme ...(3) 

[0059] wherein m and e are one of r, g, and b respectively. 

[0060] As described above, with the conventional technique based upon color 

correlation, correlation between pixel values of different color components is 
estimated near the pixel of interest, and the color component non-existent in the 
pixel of interest is generated from the color component obtained in the pixel of 
interest based upon the estimated correlation. 

[0061] However, the above-described conventional technique based upon edge 

detection has no documentation describing a method wherein the pixel with the 
non-existent color component is suitably generated in the event that judgment 
cannot be made as to the edge direction due to the difference in the vertical 
direction and the difference in the horizontal direction being the same around the 
pixel of interest X, as shown in Fig. 16C. 

[0062] Also, the conventional technique based upon the above-described color 

correlation has no documentation describing a method wherein the pixel with the 
non-existent color component is suitably generated in the event that accurate 
estimation values for color correlation cannot be obtained due to color edges being 
contained no matter how the region near the pixel of interest X is set as shown in 
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Fig. 19. 

[0063] Furthermore, the means based upon edge detection has the nature of 

exhibiting great effects for the region containing clear edges, and on the other hand, 
the means based upon color correlation has the nature of exhibiting great effects for 
the texture region. However, either conventional technique has no documentation 
describing suitable means exhibiting great effects for both the region containing 
edges and the texture region. 

[0064] Next, description will be made regarding a first embodiment according 

to the present invention. 

[0065] Fig. 1 is a block diagram which illustrates the configuration of a digital 

camera. 

[0066] The first embodiment is applying an image processing device of this 

invention to digital cameras. 

[0067] As shown in Fig. 1, the digital camera 1 comprises an optical system 2 

for focusing a luminous flux from the subject, a single-chip CCD 3 having a primary- 
Bayer-array color filter for photo-electric-conversion of the subject image formed by 
the optical system so as to output image-pickup signals, an image buffer 4 for 
temporarily storing the digitized image data output from the CCD 3, and digitized 
by an unshown A/D converting circuit or the like, a color correlation calculation 
circuit 5 which is color estimation means for estimating the color correlation near 
the pixel of interest within the image, a G generating circuit 6 for estimating the G 
component of the pixel position where the G component has been dropped out, based 
upon the color correlation estimated by the aforementioned color estimation 
calculation circuit 5 for the image data stored in the aforementioned image buffer 4 
so as to generate the G component image, an R/B generating circuit 10 for 
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constructing the R component and the B component, which have been dropped out, 
so as to generate a three-primary-color image based upon the G component image 
generated by the G generating circuit 6, the image data stored in the 
aforementioned buffer 4, and the color correlation estimated by the aforementioned 
color correlation calculation circuit 5, a color image buffer 11 for temporarily storing 
the three-primary-color image generated by the R/B generating circuit 10, an image- 
quality adjusting circuit 12 for performing image-quality adjusting processing such 
as color conversion, edge enhancement, or the like, for a color image stored in the 
color image buffer 11, a recording circuit 13 for recording the data of the three - 
primary-color image subjected to image-quality adjustment by the image-quality 
adjusting circuit, and a control circuit 14 for centrally controlling the digital camera 
including the aforementioned circuits. 

[0068] The aforementioned G generating circuit 6 comprises an combination 

average generating circuit 7 for generating averages for several combinations of G 
pixels around the pixel having no G component (pixel of interest) in the image data 
stored in the aforementioned image buffer 4, a combination selection circuit 8 which 
is combination selecting means for determining which of the combination averages 
generated by the combination average generating circuit 7 is used for the estimation 
value of the G component value non-existent in the pixel of interest based upon the 
calculation results from the aforementioned color correlation calculation circuit 5, 
and a G buffer 9 for storing a G component image obtained by generating the non- 
existent G components based upon the combinations determined by the combination 
selection circuit 8. 

[0069] Description will be made regarding the operations of the digital camera 

1 having the above-described configuration. 
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[0070] Figs. 2A and 2B are diagrams which illustrate the region near the 
pixel of interest X used for the combination average generating circuit 7 and the 
color correlation calculation circuit 5, respectively. 

[0071] Upon the user pressing an unshown shutter button, first of all, an 

optical image formed by the optical system 2 is taken by the aforementioned single- 
chip CCD 3 with the Bayer array, and an incomplete color image, wherein each 
pixel has only one color component, is stored in the image buffer 4. 
[0072] Next, the combination average generating circuit 7 performs 

processing for each pixel of the image in the image buffer 4. In this case, the 
processing differs depending upon the kind of the color component obtained at the 
pixel of interest. 

[0073] That is to say, in the event that the pixel of interest is the G pixel 

having the G component, the combination average generating circuit 7 writes the 
pixel value of the pixel of interest to the corresponding pixel position of the G buffer 
9 as it is. 

[0074] On the other hand, in the event that the pixel of interest is the R pixel 

having the R component or the B pixel having the B component, the combination 
average generating circuit 7 reads out the 3x3 pixel region centered on the pixel of 
interest (3x3 region near the pixel of interest) as shown in Fig. 2A, and calculates 
the six combination averages Vl through V6 for two G component values selected 
from the G component pixels at the upper, lower, left, and right position of the pixel 
of interest X at the center of the region, and the combination differences dl and d2 
(which are variables) for the combinations corresponding to aforementioned Vl and 
V2, as described below. 

[0075] That is to say, with the pixel at the upper position of the pixel of 
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interest X (R pixel in an example shown in Fig. 2 A) as Gl, with the pixel at the left 
position thereof as G2, with the pixel at the right position thereof as G3, with the 
pixel at the lower position thereof as G4, with the average for the upper and lower 
pixels as VI, with the average for the left and right pixels as V2, with the average 
for the upper and left pixels as V3, with the average for the right and lower pixels 
as V4, with the average for the upper and right pixels as V5, with the average for 
the left and lower pixels as V6, with the difference between the upper and lower 
pixels as dl, and with the difference between the left and right pixels as d2, the 
combination average generating circuit 7 performs calculation as shown in following 
Expression 4 and Expression 5, and outputs the results to the combination selection 



circuit 8. 




[0076] 


VI = (Gl + G4)/2 


[0077] 


V2 = (G2 + G3)/2 


[0078] 


V3 = (Gl + G2)/2 


[0079] 


V4 = (G3 + G4)/2 


[0080] 


V5 = (Gl + G3)/2 


[0081] 


V6 = (G2 + G4)/2 


[0082] 


.-.(4) 


[0083] 


dl= |G1 -G4| 


[0084] 


d2 = | G2 - G3 1 


[0085] 


....(5) 


[0086] 


While the combination average generating circuit 7 performs the 



above-described operations, the color correlation calculation circuit 5 sets the 
diamond-shaped region U formed of 25 pixels centered on the pixel of interest X as 
shown in Fig. 2B, and calculates the color correlation necessary for estimating the G 
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component of the pixel of interest in the same way disclosed in Japanese 
Unexamined Patent Application Publication No. 11-215512. Fig. 3 is a flowchart 
which indicates the color correlation estimation processing performed by the color 
correlation calculation circuit 5. 

[0087] First of all, with the color component obtained in the pixel of interest X 

as c (which indicates r, g, or b), the pixels having the color component c (i.e., the 
same color component as the pixel of interest) are specified in the region U as shown 
in Fig. 2B so as to extract the pixel values of these pixels, thereby generating the set 
Uc of the pixel values of the color component c. Furthermore, the pixels having the 
G component are specified in the region U so as to extract the pixel values of these 
pixels, thereby generating the set Ug of the pixel values of the color component G 
(Step 1). 

[0088] Next, the averages Ac and Ag, and the standard deviation Sc and Sg 

are calculated for the generated sets of the obtained pixel values Uc and Ug, 
respectively (Step 2). 

[0089] Subsequently, a and p are calculated as parameters for the color 

correlation between the color component c and the G component within the region U 
based upon the aforementioned Expression 2 as represented with the following 
Expression 6 (Step 3). 

[0090] oc = Sg/Sc, p = Ag-a- Ac ...(6) 

[0091] Subsequently, the reliability of the color correlation parameters 

calculated in Step S3 is evaluated. First of all, with the five R pixels Rl through 
R5, which are indicated by hatching within the region U near the pixel of interest as 
shown in Fig. 2B, the G components Gi non-existent in these five R pixels Rl 
through R5 are calculated from the color correlation parameters a and p using the 
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following Expression 7. 

[0092] Gi = ct-Ri + p (i = 1 through 5) ..,(7) 

[0093] Subsequently, four kinds of differences between the Gi and the G 

component at the left and right positions of the pixel of interest, and at the positions 
upper and lower the pixel of interest, are calculated for each Ri so as to obtain the 
minimal difference Ei for each Ri. 

[0094] Finally, the degree of reliability E which is a standard for the color 
correlation is obtained as the inverse number of the average of the differences Ei as 
represented with the following Expression 8. 
[0095] E = l/Avg(Ei) ...(8) 

[0096] Here, i indicates 1 through 5, and the function Avg represents 

averaging Ei. The greater the calculated degree of reliability E is, the better 
matching is obtained between the estimation result of the G component obtained 
based upon the color correlation and the G pixel values around the pixel, used for 
the estimation, and accordingly, judgment can be made that the estimation based 
upon the color correlation has succeeded (Step S4). 

[0097] The non-existent G component estimation value Xg for the pixel of 
interest is calculated based upon the value Vc of the color component c obtained in 
the pixel of interest using the following Expression 9 (Step S5), 
[0098] Xg = a-Vc+p ...(9) 

[0099] Upon the above-described color correlation estimation processing 

ending, the non-existent G component estimation value Xg and the estimation 
degree of reliability E, obtained with the color correlation estimation processing, are 
output to the combination selection circuit 8. 

[0100] At the point that the above-described processing ends, the combination 
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selection circuit 8 has obtained the information with regard to the combination 
averages VI through V6 and combination differences dl and d2 for the G pixels 
around the pixel of interest, and the non-existent G component estimation value Xg 
and the degree of reliability E thereof based upon color correlation. 
[0101] The combination selection circuit 8 generates the non-existent color 

component of the pixel of interest based upon the aforementioned information as 
shown in Figs. 4, 5A, and 5B. Fig. 4 is a flowchart which indicates the combination 
selection processing performed by the combination selection circuit 8, and Figs. 5A 
and 5B are diagrams for describing examples of the edge and the pixel value 
generated in the processing performed by the combination selection circuit 8. 
[0102] First of all, the index B for indicating the presence or absence of the 

horizontal edge or vertical edge around the pixel of interest is obtained from the 
combination differences dl and d2 using the following Expression 10 (Step 11). 
[0103] B = | dl - d2 | /(dl + d2) ...(10) 

[0104] In the event that there is a clear edge on the pixel of interest in the 

horizontal or vertical direction, e.g., in a case as shown in Fig. 5A, in general, the 
change in the pixel value is small in the direction along the edge (the vertical 
direction passing through Gl and G4 in the example shown in the drawing), and the 
change in the pixel value is great in the direction orthogonal to the edge (the 
horizontal direction passing through G2 and G3 in the example shown in the 
drawing). As a result, with the combination differences dl and d2, the difference 
between the G pixel combinations near the pixel of interest in the direction along 
the edge is small as compared with the difference between the G pixel combinations 
near the pixel of interest in the direction orthogonal to the edge. The obtained 
index B is a value for making judgment as to the presence or absence of the 
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horizontal or vertical edge near the pixel of interest X based upon the above- 
described nature, that is to say, the closer to 1 the index B is, the higher the 
probability is that a clear edge exists in the horizontal or vertical direction. 
[0105] Next, judgment is made whether or not the condition holds that the 

index B is greater than a predetermined threshold Tb and the degree of reliability E 
is less than a predetermined threshold Te (Step Si 2). 

[0106] Here, in the event that the condition does not hold, judgment is made 

that there are no clear edges in the horizontal or vertical direction, or the reliability 
for the color correlation is great, independent of the presence or absence of clear 
edges. Accordingly, six kinds of differences ej between the non-existent G 
component estimation value Xg and the combination averages Vl through V6 are 
calculated as shown in the following Expression 11. 
[0107] ej=|Xg-Vj| ...(11) 

[0108] Here, j indicates an any integer among 1 to 6. The combination 

average Vj wherein the corresponding difference ej exhibits the minimal value is 
determined to be the final non-existent G component generated value (Step S13). 
As a result, the combination average Vj which is closest to the non-existent G 
component estimation value Xg is taken as the non-existent G component generated 
value. 

[0109] On the other hand, in the event that the condition holds in the above- 

described Step S12, judgment is made that there are clear edges in the horizontal or 
vertical direction, so the reliability for the color correlation is not be excellent. In 
this case, the combination average Vj (j indicates 1 or 2) corresponding to the 
smaller one of the differences dl and d2 is determined to be the final non-existent G 
component generated value (Step SI 4). 
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[0110] Thus, upon the final non-existent G component generated value being 

obtained in Step S13 or Step S14, the combination selection processing ends. 
[0111] In general, in the event that the index B is a great value due to the 

presence of edges near the pixel of interest X in the horizontal or vertical direction, 
there is high probability that the average for the G pixels near the pixel of interest 
in the direction along the edge is closer to the true non-existent G component of the 
pixel of interest than the average for the G pixels near the pixel of interest in the 
direction orthogonal to the edge, 

[0112] In a case as shown in Fig. 5A, the difference dl = | Gl - G4 | is a small 

value, and the difference d2 = | G2 - G3 | is a great value, and accordingly, the 
combination average corresponding to the difference dl, VI = (Gl + G4)/2, is a value 
closer to the true non-existent G component of the pixel of interest X than the 
combination average V2 = (G2 + G3)/2. 

[0113] On the other hand, in the event that the region near the pixel of 

interest X does contain no edges in the horizontal or vertical direction, but contains 
edges in an oblique direction or in a texture region, the difference between the 
combination differences dl and d2 is small, and accordingly, the index B exhibits a 
small value. 

[0114] In this case, in the event that the degree of reliability E for estimation 

results for the color correlation within the region exhibits a high value, there is high 
probability that the non-existent G component estimation value Xg based upon the 
color correlation is close to the true non-existent G component of the pixel of 
interest. Note that the degree of reliability E is simply the reference for the 
reliability as to the estimation results, and accordingly, the degree of reliability E 
could be a great value even if the reliability in the estimation results is actually low. 
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In this case, in the event of using the non-existent G component estimation value 
Xg based upon the color correlation with low estimation precision for the generated 
value, the G component greatly different from the adjacent pixels is generated, and 
consequently, dot-shaped artifacts could be caused as shown in Fig. 5B. 
Accordingly, the final generated value is selected from the combination averages for 
the G pixels near the pixel of interest, so the estimation result is not greatly 
different from the pixels around the pixel of interest even in the event that the 
degree of reliability E is false to a certain degree as the index indicating the true 
reliability, and thus the dot-shaped artifacts do not readily occur. 
[0115] The above-described processing is designed based upon the above- 

described consideration so as to obtain the final optimized non-existent generated 
value even in the event that the pixel of interest is on the edge region or the texture 
region. 

[0116] Following such processing for the pixel of interest, upon the non- 

existent G component generated value being obtained, the combination selection 
circuit 8 writes the generated value to the corresponding pixel position in the G 
buffer 9. 

[0117] Upon the series of processing described above being performed by the 

combination average generating circuit 7, the color correlation calculation circuit 5, 
and the combination selection circuit 8, for the pixels of all the pixels within the 
image buffer 4, the G component image wherein non-existence of the G components 
are generated for all the pixels is obtained in the G buffer 9. 

[0118] Thus, following the G component image wherein all the pixels have the 

G components being obtained, the R/B generating circuit 10 is operated. Fig. 6 is a 
flowchart which indicates the R/B generating processing performed by the R/B 
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generating circuit 10. 

[0119] Upon the R/B generating processing being started, a region with a 
predetermined size near the pixel of interest is read out for each pixel X of an image 
stored in the image buffer 4, and the G components are read out from the region 
near the corresponding pixel of interest stored in the aforementioned G buffer 9 
(Step S21). The size of the region read out differs according to the kind of the color 
component of the pixel of interest X, that is to say, a region of 3 x 3 pixels is read 
out for the pixel having the R component or the B component, and a region of 4 x 4 
pixels is read out for the pixel having the G component. 

[0120] The data arrangements read out in the aforementioned cases are 

shown in Figs. 7A and 7B. Figs. 7A and 7B are diagrams which illustrates 
examples of the regions taken near the pixel of interest in the processing performed 
by the R/B generating circuit 10. 

[0121] Fig. 7 A illustrates an example in the event that the pixel of interest X 
has the color component B (B5 in the example shown in the drawing) obtained by 
the CCD 3 in image-taking, and at the point that processing is performed by the R/B 
generating circuit 10, the pixel of interest also has the color component G5 
generated by the above-described G generating circuit 6. Note that in the event 
that the pixel of interest X has the color component R obtained by the CCD 3 in 
image-taking, the data arrangement is modified so as to interchange the B 
component with the R component in Fig. 7A. 

[0122] On the other hand, Fig. 7B illustrates an example in the event that the 

pixel of interest X has the color component G (G6 in the example shown in the 
drawing) obtained by the CCD 3 in image-taking, and the region of 4 x 4 pixels near 
the pixel of interest is employed, which is somewhat large as compared with the 
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case shown in Fig. 7A as described above. This is because in the event that a region 
of 3 x 3 pixels is set for the pixel having the G component obtained from the CCD 3, 
the region contains only two each of pixels having the R components and the B 
components. 

[0123] Next, with the color component non-existent in the pixel of interest X 

being as the color component c, the color correlation between the color component c 
and the G component is estimated within the region read out. The G components 
are obtained for all the pixels within the region near the pixel of interest in the 
processing performed by the above-described G generating circuit 6, and 
accordingly, two kinds of the color components, i.e., the G components and the color 
components c, are obtained in the pixel positions having the color components c near 
the pixel of interest X, as shown in Figs. 7A and 7B. Accordingly, in these pixel 
positions, with the G components as the data Y, and with the pixel values of the 
color components c as the data X, and making an assumption that the 
approximation relation Z = etc • Y + pc holds, the parameters for the relation ac and 
P are calculated with the known least square method. Furthermore, in the event 
that the color component obtained in the pixel of interest X is the G component, two 
kinds of the components, i.e., the R component and the B component are non- 
existent, so the above-described estimation is performed two times for the R 
component and the B component, thereby calculating the parameters ar and pr, and 
ab and pb (Step S22). 

[0124] The computation (ac • Vg + pc) is performed using the approximation 

parameters ac and pc thus calculated, and the G component Vg of the pixel of the 
interest X, thereby estimating the pixel value of the color component c non-existent 
in the pixel of interest. At this time, in the event that the color component obtained 
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in the pixel of interest is the G component, the computation is performed two times 
for the R component and the B component as to the c component shown in the 
aforementioned Expression (Step S23). 

[0125] Upon such processing being performed for all the pixels of the image 

stored in the image buffer 4, the R/B generating processing ends, thereby obtaining 
tri-color components for all the pixels wherein one color component is the component 
obtained in image-taking, and the other two color components are the estimated 
components. The tri-color image thus obtained is stored in the color image buffer 
11. 

[0126] The color image stored in the color image buffer 11 is subjected to color 

conversion, contrast adjustment, edge enhancement, or the like, by the image 
quality adjusting circuit 12, and subsequently, is compressed by the recording 
circuit 13 so as to record on a recording medium or the like. 
[0127] Note that the present embodiment is not restricted to the above- 

described arrangement, but rather, various modifications may be made. 
[0128] For example, while description has been made regarding the 

arrangement wherein the combination averages are obtained only for the G 
component, and following the all the non-existent G components being generated, 
the R components and B components are obtained based upon color correlation, an 
arrangement may be made wherein the non-existent color components are 
generated by selecting from the combination averages near the pixel of interest for 
the R components and the B components in the same way as with the G 
components. 

[0129] Furthermore, while description has been made regarding the 

arrangement wherein the combination average is obtained from the combination of 
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two pixels from the pixels near the pixel of interest, an arrangement may be made 
wherein the combination averages and the combination differences are obtained 
from the combination of three or more pixels from the pixels near the pixel of 
interest. 

[0130] Furthermore, while description has been made regarding the 

arrangement wherein processing is performed by the hardware inside the digital 
camera 1 serving as an image processing device, an arrangement easily can be 
made wherein such processing is performed on a computer such as a PC (personal 
computer) or the like with an image processing program. Fig. 8 is a flowchart which 
indicates software processing performed by the computer. 
[0131] With the software processing performed by the image processing 

program, an image Inlmg having only one color component at each pixel is input, 
and a tri-color image Outlmg is generated and output. Let us say that these 
memory regions have been prepared for the processing beforehand. At this time, 
the memory region for the Inlmg corresponds to the image buffer 4 in. the hardware 
shown in Fig. 1, and the memory region for the Outlmg corresponds to the color 
image buffer 11 in the hardware shown in Fig. 1. 

[0132] Upon the processing being started, first of all, the memory region for 

using as the buffer GImg for generating G component (corresponding to the G buffer 
9 in the hardware shown in Fig. 1) is allocated. Subsequently, the pixel values of 
the pixels having the G components in the image Inlmg are copied to the 
corresponding pixel positions in the GImg without change (Step S31). 
[0133] Next, one of the unprocessed pixels having the R component or the B 

component in the image Inlmg is selected as the pixel of interest X (Step S32). 
[0134] Subsequently, the combination averages Vl through V6, and the 
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combination differences dl and d2 corresponding VI and V2 are calculated for the 
pixels having the G component contained in the region of 3 x 3 pixels near the pixel 
of interest, as shown in Fig. 2A described above (Step S33). 

[0135] Subsequently, the region U near the pixel of interest is set as shown in 

Fig. 2B described above, and the color correlation estimation processing is 
performed as shown in Fig. 3 described above, thereby calculating the non-existent 
G component estimation value Xg and the degree of reliability E for the estimation 
based upon color correlation estimation (Step S34). 

[0136] Subsequently, the combination selection processing as shown in Fig. 4 

described above is performed, and the non-existent G component generated value is 
calculated for the pixel of interest, whereby the calculation result is written to the 
corresponding pixel position in the GImg (Step S35). 

[0137] Judgment is made as to whether there are any unprocessed pixels 
having the R component or the B component in the image Inlmg (Step S36), and in 
the event that there are any, the flow returns to the processing in Step S32 
described above, and the processing as described above is repeatedly performed. 
[0138] On the other hand, in the event that there are no unprocessed pixels, 

the R/B generating processing as described in Fig. 6 is performed based upon the 
GImg and Inlmg so as to generate the non-existent color component other than the 
G component for each pixel, and the generated result is written to the corresponding 
pixel position in the Outlmg (Step S37), whereby the processing ends. 
[0139] Note that while description has been made regarding an arrangement 

employing a single-sensor image-pickup system including primary-color Bayer-array 
color filters, an arrangement may be made wherein a single-sensor image-pickup 
system including complementary-color-Bayer-array color filters or other color filters 
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are employed. Furthermore, it is needless to say that the above-described 
configuration can be applied to an arrangement wherein a digital image obtained 
from the two-sensor image-pickup system or the triple-sensor pixel spatial offset 
image-pickup system wherein one or more color components are non-existent for 
each pixel is subjected to estimation for the non-existent components for each pixel 
so as to be output as a color digital image. 

[0140] In a case that the reliability for the color correlation estimation results 

is low, in the event that the non-existent color component estimation is performed 
based upon the estimation results, dot-shaped artifacts could occur. However, with 
the first embodiment described above, the plural combination averages for the pixel 
values near the pixel of interest are generated by the combination average 
generating circuit 7 for multiple combinations, one value is selected from these 
combination averages by the combination selection circuit 8 based upon the color 
correlation estimation results so as to be determined to be the non-existent color 
component, and accordingly, matching is improved between the generated non- 
existent component in the pixel of interest and the pixels therearound, and thus the 
artifacts are hardly caused. 

[0141] Furthermore, with the present embodiment, in the event that the 

reliability for the color correlation estimation results is great, the combination 
average is selected based upon the estimated color correlation, the non-existent 
color component can be generated with high precision as compared with simple 
linear interpolation or the like. 

[0142] Furthermore, with the present embodiment, the reliability for the color 

correlation estimation results is evaluated, and in the event that the reliability is 
great, the combination average close to the non-existent component candidate 
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calculated based upon color correlation is selected, and thus, the non-existent color 
component can be estimated with excellent precision in the texture region or the 
like. 

[0143] On the other hand, in the event that the reliability for the color 

correlation estimation results is low, the combination average corresponding to the 
combination wherein the minimal fluctuation has been calculated by the 
combination average generating circuit 7 is selected as the non-existent color 
component by the combination selection circuit 8, thereby enabling the non-existent 
component to be estimated with excellent precision even for the edge region which 
reduces the reliability for the color correlation estimation results. 
[0144] Next, description will be made regarding a second embodiment 

according to the present invention. 

[0145] Fig. 9 is a block diagram which illustrates a configuration of a digital 

camera of a second embodiment. 

[0146] With the second embodiment, description will be omitted with regard 

to the same components as with the above-described first embodiment, the same 
component are denoted with the same reference characters as with the first 
embodiment, and description will be primarily made only regarding the differences. 
[0147] With this second embodiment, the image processing device of the 

present invention is applied to the digital camera in the same way as with the 
above-described first embodiment. 

[0148] A digital camera 21 of the present second embodiment further 

comprises a region judgment circuit 24 serving as evaluation means and also region 
judgment means as compared with the digital camera 1 according to the above- 
described first embodiment, and so the operations are different in a combination 
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average generating circuit 27 serving as a first non-existent color component 
estimation means in a G generating circuit 26, a color correlation calculation circuit 
25 serving as a second non-existent color estimation means, and a combination 
selection circuit 28 serving as a third non-existent color estimation means in the G 
generating circuit 26. 

[0149] The operations of the entire digital camera 21, from an unshown 

shutter button being pressed by the user up to the obtained image in the single - 
plate state being stored in the image buffer 4, are the same as with the above- 
described first embodiment. 

[0150] Next, the combination average generating circuit 27 performs 

processing for each pixel of the image in the image buffer 4. Here, in the event that 
the pixel of interest has the color component G, the processing for the pixel of 
interest is the same as with the above-described first embodiment. 
[0151] On the other hand, in the event that the pixel of interest has the R 

component or B component, the operations are somewhat different from those of the 
above-described first embodiment. That is to say, the combination average 
generating circuit 27 reads out a 3 x 3 region near the pixel of interest, calculates 
six kinds of the combination averages VI through V6, and the differences between 
the corresponding combinations dl through d6 for the G components obtained at the 
left and right positions of the pixel of interest X which is a center of the region, and 
at the upper and lower positions of the pixel of interest, as shown in Fig. 10, and 
outputs these results to the combination selection circuit 28. 
[0152] The processing performed in parallel with the above-described 

processing performed by the combination average generating circuit 27 is somewhat 
different from that of the above-described first embodiment. 
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[0153] First of all, following the flowchart as shown in Fig. 12, the region 

judgment circuit 24 judges the type of the region, i.e., whether or not the region of 
interest is an edge region, or whether or not the region of interest is a texture 
region, and calculates the predicted degree of reliability E for color correlation 
estimation in the region, according to the type of the region. 
[0154] Description will be made below regarding each step shown in the 

flowchart in Fig. 12 with reference to Figs. 11 A, 11B, 11C, or the like, as necessary. 
Figs. 11 A, 11B, and 11C are diagrams which illustrate the region near the pixel of 
interest used by the region judgment circuit 24, and Fig. 12 is a flowchart which 
indicates region judgment processing performed by the region judgment circuit 24. 
[0155] As shown in Fig. 11 A, a 7 x 7 pixel region centered on the pixel of 

interest is taken around the pixel of interest X, and furthermore, a 4 x 4 pixel sub- 
region is taken inside the 7x7 pixel region as indicated with the bold frame shown 
in Fig. 11A (Step S41). Sixteen kinds of the sub-regions can be taken according to 
the pixel position selected to be the pixel of interest X within the 4x4 pixel sub- 
region. Accordingly, let us say that the upper-left pixels of the sixteen kinds of sub- 
regions, which can be taken for the 7x7 regions, are denoted by the reference 
numerals 1 through 16 as shown in Fig. 11B, and the sub-regions will be referred to 
as Ul through U16 using the reference numerals. For example, in Fig. 11B, the 
upper-left pixel of the sub-region is at the seventh position, so the sub-region will be 
referred to as "sub-region U7". 

[0156] The standard deviation ak of the G components obtained within the 

sub-region is calculated for each sub-region Uk (k is 1 through 16), and the minimal 
value min and the maximal value max for the standard deviation ak are calculated 
(Step S42). 
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[0157] Subsequently, the type of the region near the pixel of interest is 

classified based upon these minimal value min and maximal value max as shown in 
following Steps S43 and S44. Note that Tl through T3 shown in Steps S43 and S44 
are predetermined thresholds. 

[0158] With classification of the type of the region, first of all, judgment is 

made whether or not the relations min < Tl and max - min < T2 hold with regard to 
the minimal value min and the maximal value max obtained in the above-described 
Step S42 (Step S43). 

[0159] In the event that the relations hold, judgment is made that the region 

is uniform, and the degree of reliability E is set to 0 (Step S45), whereby the 
processing ends. 

[0160] Conversely, in the event that the relations do not hold in the 

aforementioned Step S43, judgment is further made whether or not the relations 
min < Tl and max - min > T3 hold (Step S44). 

[0161] In the event that the relations holds in Step S44, judgment is made 

that the region is an edge region, and the reliability is set to 0 (Step S46), whereby 
the processing ends. 

[0162] Conversely, in the event that the relations do not hold in the above- 

described Step S44, judgment is made that the region is a texture region. In this 

* 

case, first of all, each sub-region Uk is classified based upon the condition whether 
or not the relation ak - min < T4 holds. Here, T4 represents a predetermined 
threshold, and in the event that the condition is satisfied, the sub-region Uk can be 
regarded as being relatively uniform. Subsequently, the union IT of the sub-regions 
Uk satisfying the condition is generated. 

[0163] An example of the U' thus generated is shown in the hatched region in 
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Fig. 11C. Furthermore, the degree of reliability E is calculated using (min + T4) 
which indicates the maximal value of the standard deviation for the U f as 
represented by the following Expression 12 (Step S47), whereby the processing 
ends. 

[0164] E = l/(min + T4) ...(12) 

[0165] Following such region judgement processing being performed, the 

region judgment circuit 24 outputs the degree of reliability E to the combination 
selection circuit 28, and in the event that the degree of reliability E is not 0, the 
region judgment circuit 24 further outputs the coordinates of the pixels contained in 
the U' generated in Step S47, to the color correlation calculation circuit 25. 
Conversely, in the event that the degree of reliability E is 0, the region judgment 
circuit 24 does not output the coordinates to the color correlation calculation circuit 
25, and estimation of the color correlation is not performed for the pixel of interest 
X. 

[0166] In the event that the coordinates are input from the region judgment 

circuit 24, the color correlation calculation circuit 25 estimates the color correlation 
necessary for generating the G component of the pixel of interest in the same way as 
with the above-described first embodiment. Note that the shape of the region near 
the pixel of interest is not restricted to the shape as shown in Fig. 2B described 
above, and evaluation of the reliability for the color correlation is not performed, 
unlike the above-described first embodiment. 

[0167] Fig. 13 is a flowchart which indicates color correlation estimation 

processing performed by the color correlation calculation circuit 25. 
[0168] Upon the color correlation estimation processing being started, the 

pixel sets Uc and Ug are extracted from the pixel set U' near the pixel of interest 
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(Step S51). The region used at this time is not the region U indicated in Fig. 2B 
described above, but the region IT formed of coordinates specified by the region 
judgment circuit 24, unlike the processing in step Si shown in Fig. 3, and other 
processing is the same as with Step Si. 

[0169] The subsequent processing of Steps S52, S53, and S54, are the same as 

with Steps S2, S3, and S5, in Fig. 3 for the above-described first embodiment, 
respectively. Upon the processing in Step S54 ending, the color correlation 
estimation processing ends. 

[0170] Upon the color correlation estimation processing ending, the color 

correlation calculation circuit 25 outputs only the non-existent G component 
estimation values Xg, to the combination selection circuit 28. 

[0171] The degree of reliability E is input to the combination selection circuit 

28 from the region judgment circuit 24, not from the color correlation calculation 
circuit 25, unlike the above-described first embodiment. Fig. 14 indicates 
operations for the pixel of interest following data being input from the combination 
average generating circuit 27 and the color correlation calculation circuit 25. 
[0172] Fig. 14 is a flowchart which illustrates combination selection 

processing performed by the combination selection circuit 28. 
[0173] First of all, the index j (which is an integer between 1 to 6), wherein 

the combination difference dj exhibits the minimal value of dl through d6, is taken 
as minj, and the combination average Vminj is taken as the first non-existent G 
component generated value XI (Step S61). 

[0174] Next, judgment is made whether or not the degree of reliability E is 0 

(Step S62), in the event that the degree of reliability E is 0, the obtained XI is 
determined to be the final non-existent G component generated value (Step S63). 
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[0175] Conversely, in the event that the degree of reliability E is not 0 in the 

above-described Step S62, the weighted average is calculated for XI and the non- 
existent component estimation value Xg based upon the degree of reliability E 
thereof, thereby obtaining the final non-existent generated value X2, as represented 
with the following Expression 13 (Step S64). 
[0176] X2 = (XI + E ■ Xg)/(1 + E) ...(13) 

[0177] Upon the combination selection circuit 28 obtaining the final non- 

existent G component generated value for the pixel of interest X following the 
processing ending, the combination selection circuit 28 writes the generated result 
to the corresponding address in the G buffer 9. 

[0178] The subsequent operations performed by other circuits are the same as 

with the above-described first embodiment. 

[0179] Note that with the present embodiment, various modifications may be 

made, as well. For example, an arrangement may be made wherein the judgment 
whether or not the region is a texture region by the region judgment circuit 24 is 
performed using known texture analysis means. 

[0180] Furthermore, while description has been made regarding the 

arrangement wherein processing is performed by the internal hardware in the 
digital camera 21 serving as an image processing device, an arrangement may be 
easily made wherein such processing is performed by an image processing program 
on a computer such as a PC (personal computer). Figs. 15A and 15B are flowcharts 
which illustrate the software processing performed by the computer. 
[0181] With the software processing performed by the image processing 

program, an image Inlmg having one color component at each pixel is input, and a 
tri-color image Outlmg is generated and output, in the same way as with the 
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software processing of the first embodiment described above. 

[0182] The processing shown in Steps S71, S72, S78, and S79, in the 

flowchart, is the same as the processing shown in Steps S31, S32, S36, and S37, 

respectively, so description will be made only regarding the processing in other 

Steps. 

[0183] Upon the processing in Step S72 ending, the combination averages VI 

through V6, and the combination differences dl through d6, for the pixels having 
the G components within the region of 3 x 3 pixels near the pixel of interest X are 
calculated, as shown in Fig. 10 (Step S73). 

[0184] Subsequently, the region judgment processing as shown in Fig. 12 

described above is performed so as to judge the type of the region, thereby 
calculating the predicted degree of reliability E in case of the estimation of the color 
correlation within the region and calculating the sub -region U f with relatively high 
uniformity within the region (Step S74). 

[0185] Subsequently, judgment is made whether or not the degree of 

reliability E is 0 (step S75), and in the event that the degree of reliability E is not 0, 
the color correlation estimation processing as shown in Fig. 13 described above is 
performed for the region U around the pixel of interest, set in Step S74, thereby 
calculating the non-existent G component estimation value Xg based upon the color 
correlation estimation (Step S76). 

[0186] Conversely, in the event that the degree of reliability E is 0 in Step S75 

described above, or the processing in Step S76 described above ends, the 
combination selection processing shown in Fig. 14 described above is performed so 
as to calculate the non-existent G component generated value for the pixel of 
interest, and writes to the corresponding pixel position in GImg (Step S77), whereby 
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the flow proceeds to the above-described Step S78. 

[0187] With the above-described second embodiment, the same general effects 

can be obtained as with the above-described first embodiment, and also, based upon 
the evaluation results by the region judgment circuit 24 for the degree of reliability 
of the color correlation, the combination selection circuit 28 calculates the weights 
for the combination average obtained by the combination average generating circuit 
27 and for the non-existent color component estimation value obtained by the color 
estimation calculation circuit 25 based upon color correlation, thereby enabling the 
non-existent color component to be estimated with high precision without artifacts 
regardless of the reliability of the color correlation. 

[0188] Furthermore, judgment whether or not the region near the pixel of 

interest is a texture is employed by the region judgment circuit 24 for the evaluation 
standard for the evaluation means. Accordingly, in the event that the region is a 
texture region wherein the non-existent color component can be generated with high 
precision based upon color correlation, the estimation based upon the color 
correlation has larger effect on generating the non-existent value. Conversely, in 
the event that the region is an edge region wherein the non-existent color 
component can be generated with low precision based upon color correlation, the 
estimation based upon the color correlation has smaller effect on generating the 
non-existent value. Thus, with the present embodiment, non-existent color 
components can be estimated with excellent precision for any type of the region near 
the pixel of interest. 

[0189] As described above, with the image processing device and the image 

processing program according to the present invention, the non-existent component 
of each pixel of a digital image wherein one or more color components are non- 
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existent in each pixel can be more suitably estimated, thereby generating a color 
digital image. 

[0190] Note that the present invention also encompasses modifications and 

the like, configured by partially combining the above-described embodiments or the 
like, as well. 

[0191] In this invention, it is apparent that working modes different in a wide 

range can be formed on this basis of this invention without departing from the spirit 
and scope of the invention. This invention is not restricted by any specific 
embodiment except being limited by the appended claims. 

* * * 



-34- 



